{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Python基础和Numpy基础\n",
    "\n",
    "主要参考阿里天池的[Pandas实践-第一章 预备知识](https://tianchi.aliyun.com/course/325/3660)部分。"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 1.Python基础记录\n",
    "记录以下一些理解难度稍微大一些的知识点"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 1.1 zip对象、 enumerate、 如何理解\\*和**\n",
    "参考[Python中的*（星号）和**(双星号）完全详解](https://blog.csdn.net/zkk9527/article/details/88675129?utm_medium=distribute.pc_relevant_t0.none-task-blog-2%7Edefault%7EBlogCommendFromMachineLearnPai2%7Edefault-1.control&dist_request_id=1330144.22821.16181295666102549&depth_1-utm_source=distribute.pc_relevant_t0.none-task-blog-2%7Edefault%7EBlogCommendFromMachineLearnPai2%7Edefault-1.control)\n",
    "\n",
    " `*`和`**`在函数声明和调用的时候的意义是不一样的，我觉得这和java中形参的**可变参数**的定义(如String... args) 很像，这样做的好处是可以传递多个参数\n",
    "\n",
    " 但是这里会将形参转换为一个**元组**或者**字典**！！！（关键）"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 2 [3, 4, 5, 6]\n",
      "myprint1: (1, 2, 3)\n",
      "myprint2: {'x': 1, 'y': 2, 'z': 3}\n",
      "myprint3: 1 2 (3, 4)\n",
      "myprint4: 1 2 3\n"
     ]
    }
   ],
   "source": [
    "from typing import List\n",
    "\n",
    "a,b,*c=[1,2,3,4,5,6] # c会收集多余的参数\n",
    "print(a,b,c)\n",
    "\n",
    "# params将会是元组\n",
    "def myprint1(*params):\n",
    "    print('myprint1:', params)\n",
    "myprint1(1,2,3)\n",
    "\n",
    "# params将会是字典\n",
    "def myprint2(**params):\n",
    "    print('myprint2:', params)\n",
    "myprint2(x=1,y=2,z=3)\n",
    "\n",
    "# 如果形参加*和实参加*是两个相反的过程\n",
    "\n",
    "def myprint3(x,y,*z):\n",
    "    print('myprint3:', x,y,z)\n",
    "param=[1,2,3,4] # 列表或元组类型的参数\n",
    "myprint3(*param) # 或*[1,2,3,4]\n",
    "\n",
    "def myprint4(x,y,z):\n",
    "    print('myprint4:', x,y,z)\n",
    "param={'x':1,'y':2,'z':3} # 字典类型参数\n",
    "myprint4(**param)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "outputs": [
    {
     "data": {
      "text/plain": "[('a', 'd', 'h'), ('b', 'e', 'i'), ('c', 'f', 'j')]"
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 练习zip和enumerate\n",
    "L1, L2, L3 = list('abc'), list('def'), list('hij')\n",
    "\n",
    "zipped = list(zip(L1,L2,L3))\n",
    "zipped"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "outputs": [
    {
     "data": {
      "text/plain": "[(0, ('a', 'd', 'h')), (1, ('b', 'e', 'i')), (2, ('c', 'f', 'j'))]"
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enum = list(enumerate(zipped))\n",
    "enum"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('a', 'b', 'c'), ('d', 'e', 'f'), ('h', 'i', 'j')]\n"
     ]
    }
   ],
   "source": [
    "# 还原L1 L2 L3\n",
    "print(list(zip(*zipped)))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 2.Numpy基础记录\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "outputs": [
    {
     "data": {
      "text/plain": "array([[[0.47185957, 0.84477606, 0.98059455],\n        [0.2719314 , 0.33803042, 0.68810287]]])"
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "np.random.rand(1,2,3)  # rand()的参数是可变参数，输出的随机数服从0-1均匀分布"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### random对象\n",
    "以下都是random对象常用的方法："
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "outputs": [
    {
     "data": {
      "text/plain": "array([[13.28416483,  6.21784826,  3.16208642],\n       [12.23946997,  9.00966496,  4.69854172]])"
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.uniform(3,15,(2,3))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "注意下面的`randn()`，该方法的**形参**是有`*`符号的，类似于Java的`int... args`语法糖，也要注意**实参**若有`*`符号如何理解"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "outputs": [
    {
     "data": {
      "text/plain": "array([[ 0.03958001, -1.29166956,  0.20148813],\n       [-0.27772136, -2.05007173, -1.27403723]])"
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.randn(2,3)  # 服从标准正态分布"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "outputs": [
    {
     "data": {
      "text/plain": "array([[4, 2, 6],\n       [7, 9, 7]])"
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.randint(1,10,(2,3)) #low和high是左闭右开哦"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "outputs": [
    {
     "data": {
      "text/plain": "array([1, 2, 3])"
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array({'a':1,'b':2})\n",
    "np.array((1,2,3))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### reshape()\n",
    "感觉是经常用到的，但是要注意一点，这让我花了很多时间才搞明白\n",
    ">Returns an array containing the same data with a new shape.\n",
    "\n",
    "也就是说**返回的**新数组才具有新的shape，原数组的shape不会有变化"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3 4]\n",
      " [6 7 8 9]] \n",
      "\n",
      "[[1 2]\n",
      " [3 4]\n",
      " [6 7]\n",
      " [8 9]]\n"
     ]
    }
   ],
   "source": [
    "target = np.array([\n",
    "    [1,2,3,4],\n",
    "    [6,7,8,9],\n",
    "])\n",
    "print(target,'\\n')\n",
    "target = target.reshape((4,-1))  # 注意这里！返回的array才具有新的shape\n",
    "print(target)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 3. 练习"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Ex1：利用列表推导式写矩阵乘法\n",
    "\n",
    "很有意思的一道题，其实可以由内而外的构造，即先构造外层的i，再j，最后再k"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "outputs": [
    {
     "data": {
      "text/plain": "True"
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M1 = np.random.rand(2,3)\n",
    "M2 = np.random.rand(3,4)\n",
    "item=0\n",
    "res = [[sum([M1[i][k]*M2[k][j] for k in range(M1.shape[1])]) for j in range(M2.shape[1])] for i in range(M1.shape[0])]\n",
    "(np.abs((M1@M2 - res) < 1e-15)).all() # 排除数值误差"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Ex2：更新矩阵\n",
    "\n",
    "考察广播运用，自己总结的话是，**二维数组**能够广播的条件是：假设A=(m ,n)，矩阵B能够与之进行广播的条件是B的维度为`(m ,1)`或`(1,n)`。\n",
    "\n",
    "在下面的Ex3可以发现，其实**列向量和行向量**也可以广播，准确说应该是**相互**广播\n",
    "\n",
    "参考：[numpy的广播(Broadcasting)](https://zhuanlan.zhihu.com/p/35010592)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "outputs": [
    {
     "data": {
      "text/plain": "array([[ 2.82896825,  5.65793651,  8.48690476],\n       [11.31587302, 14.14484127, 16.97380952],\n       [19.80277778, 22.63174603, 25.46071429]])"
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A=np.arange(1,10).reshape(3,-1)\n",
    "B=A*(1/A).sum().reshape(-1,1)\n",
    "B"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "outputs": [
    {
     "data": {
      "text/plain": "array([[2., 2., 2.],\n       [2., 2., 2.]])"
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A*(1/A*2)  # 尝试广播机制\n",
    "\n",
    "np.ones(3)+np.ones((2,1))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Ex3：卡方统计量\n",
    "\n",
    "`sum()`函数的`axis`参数其实还是很好理解，比如这里的A是一个(8, 5)的张量，那么`axis=0`表示取张量为0这个维度，对于二维张量也就是矩阵来说，按列求和。\n",
    "\n",
    "这里B的计算方式确实很难理解啊，`A.sum(axis=0)`计算出的结果是一个(1, 5)的张量，而`A.sum(axis=1)`计算出来的是一个**(1, 8)**的张量。发现没，两个结果都会是**行向量！**，我去！！！但是明明`A.sum(1)`应该是个列向量嘛，但没办法，该方法计算出来统一都是行向量。\n",
    "\n",
    "所以到了这一步，需要将`A.sum(1)`进行`reshape()`，变成一个列向量(8, 1)，这样才可以和`A.sum(0)`进行广播，得到一个(8, 5)的张量，然后此时A.sum()是一个标量\n",
    "\n",
    "注意这里有个很神奇的现象哦，(5,1)和(1,8)也可以广播，但这里的广播应该算是一种特殊的广播，我将其命名为**互相广播**。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "outputs": [
    {
     "data": {
      "text/plain": "11.842696601945802"
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "A = np.random.randint(10, 20, (8, 5))\n",
    "\n",
    "# print(A.sum(0).reshape(-1,1).shape)\n",
    "# print(A.sum(1).shape)\n",
    "# print(A.sum(0).reshape(-1,1)*A.sum(1))\n",
    "\n",
    "B = A.sum(axis=1).reshape(-1,1)*A.sum(axis=0)/A.sum()  # 这里一定得reshape，不然没有办法\n",
    "K = (((A-B)**2)/B).sum()\n",
    "K"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "outputs": [
    {
     "data": {
      "text/plain": "array([[0.11098894, 0.7313945 , 0.40403255, 0.32104524, 0.20482787],\n       [0.06539065, 0.43091104, 0.23804129, 0.18914818, 0.12067713],\n       [0.08148688, 0.53698188, 0.2966363 , 0.23570792, 0.15038239],\n       [0.00269358, 0.01775014, 0.00980543, 0.00779142, 0.00497095],\n       [0.08854008, 0.58346106, 0.32231205, 0.25610993, 0.16339894],\n       [0.08774593, 0.5782278 , 0.31942113, 0.2538128 , 0.16193336],\n       [0.08843952, 0.58279837, 0.32194597, 0.25581905, 0.16321335],\n       [0.13528939, 0.89152948, 0.49249335, 0.39133641, 0.24967385]])"
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 这是我上面提到的互相广播，适用于列向量和行向量之间的运算\n",
    "np.random.rand(1,5)*np.random.rand(8,1)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Ex4：改进矩阵计算的性能\n",
    "妈的看不懂看不懂...主要是这个公式很难理解啊：\n",
    "\\begin{aligned}\n",
    "Y_{i j} &=\\left\\|B_{i}-U_{j}\\right\\|_{2}^{2} \\\\\n",
    "&=\\sum_{k=1}^{p}\\left(B_{i k}-U_{k j}\\right)^{2} \\\\\n",
    "&=\\sum_{k=1}^{p} B_{i k}^{2}+\\sum_{k=1}^{p} U_{k j}^{2}-2 \\sum_{k=1}^{p} B_{i k} U_{k j}\n",
    "\\end{aligned}\n",
    "\n",
    "我过了一天再来看倒是看懂了，我觉得难点有两个：\n",
    "\n",
    "一个是$$Y_{ij}$$如何表示。将$$||B_i-U_j||_2^2$$表达为$$Y_{ij}$$就很难想到了，我过后发现我思维受阻的地方在于，我表达成了$$Y$$而非$$Y_{ij}$$，当然现在看起来是很幼稚的错误，但是当时思考的时候没有注意到公式$$R$$中其实是有$$\\sum_{i=1}^m\\sum_{j=1}^n$$。\n",
    "\n",
    "另一个是理解$$B_i$$和$$U_j$$相减，两个其实都是p维向量！！，既然如此，又根据$$||a||_2^2=\\sum_{i=1}^{n}a_i^2$$，其实是对n维向量各个元素的平方进行累加的，那既然$$B_i-U_j$$是p维的，很自然就能得到$$\\sum_{k=1}^p(B_{ik}-U_{kj})^2$$。接下来的展开很简单，不赘述。\n",
    "\n",
    "现在发现这真的是个很经典的题，值得反复把玩😍"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "outputs": [
    {
     "data": {
      "text/plain": "100566"
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "m, n, p = 100, 80, 50\n",
    "B = np.random.randint(0, 2, (m, p))\n",
    "U = np.random.randint(0, 2, (p, n))\n",
    "Z = np.random.randint(0, 2, (m, n))\n",
    "\n",
    "(((B**2).sum(axis=1).reshape(-1,1)+(U**2).sum(axis=0)-2*B@U)*Z).sum()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Ex5：连续整数的最大长度\n",
    "\n",
    "有亿点点难...\n",
    "\n",
    "注意这里的max()是**numpy对象**的方法，当`axis`参数不赋值时是对**全局**取最大值，和`sum()`类似"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[False  True False False]\n",
      "[1 0 1 0 0 1]\n",
      "(array([0, 2, 5], dtype=int64),)\n",
      "[[2 3]]\n"
     ]
    },
    {
     "data": {
      "text/plain": "3"
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = [1,2,5,6,7]\n",
    "print(np.diff(x)!=1) # 如果为0，表示当前元素与前一个元素是连续的\n",
    "print(np.r_[1,np.diff(x)!=1,1]) # 加上边界，很明显边界应该是1，也就是不连续的\n",
    "print(np.nonzero(np.r_[1,np.diff(x)!=1,1]))  #拿到为1的索引，也就是连续递增断开的地方\n",
    "print(np.diff(np.nonzero(np.r_[1,np.diff(x)!=1,1]))) # 找到每个断开的地方的长度\n",
    "np.diff(np.nonzero(np.r_[1,np.diff(x)!=1,1])).max()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "outputs": [
    {
     "data": {
      "text/plain": "3"
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 法2 可以封装成一个匿名函数\n",
    "f = lambda i : np.diff(np.nonzero(np.r_[1,np.diff(i)!=1,1])).max()\n",
    "f([1,2,3,3,6,7,8])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### 附加思考题\n",
    "考虑一下如何用numpy表达公式：\n",
    "$$d((y_1,\\dots,y_n),(z_1,\\dots,z_n)) = \\sqrt{\\sum_{j=1}^{n}(\\frac{y_j} {M_j} - \\frac{z_j} {M_j})^2}$$\n",
    "\n",
    "来源：[一文搞懂k近邻（k-NN）算法（一）](https://zhuanlan.zhihu.com/p/25994179)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "outputs": [
    {
     "data": {
      "text/plain": "array([9, 7, 8, 6, 9, 5, 8, 7, 9, 9, 9, 9])"
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = 10\n",
    "n = 12\n",
    "X = np.random.randint(10,size=(m,n))\n",
    "M = X.max(axis=0)-X.min(axis=0)\n",
    "M"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "outputs": [
    {
     "data": {
      "text/plain": "1.6601788863693627"
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 取任意两行计算欧氏距离\n",
    "y = X[1]\n",
    "z = X[3]\n",
    "\n",
    "d = (((y/M-z/M)**2).sum())**1/2  # 欧氏距离\n",
    "d"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}